• Setup
• How to profile code
  • Look for “slow” sections
• Making improvements
• Single node parallel
  • Multiple cpus, 1 computer
• Multi node parallel
  • Multiple cpus, multiple computers

• Slides are available at https://uschpc.github.io/workshop-hpc-python/
  
• git clone https://github.com/uschpc/workshop-hpc-python.git
• Open index.html in browser
• examples in examples
• Requirements:
  • python3
  • mpi implementation
    • mpi4py
  • hdf5
    • h5py (MPI parallel enabled)
  • snakeviz
  • viztracer
  • Most available on CARC clusters

snakeviz (Must run this on local computer)

pip install snakeviz

viztracer

pip install viztracer

h5py (optional)

module load usc
module load python
module load hdf5
CC="mpicc" HDF5_MPI="ON" HDF5_DIR=$HDF5_ROOT pip3 install --no-binary=h5py h5py --user

• Aim for fast enough code
  
• Compute time is less expensive than human time

1. Profile code to understand the execution time and memory use of each part
2. Identify bottlenecks (i.e., parts of code that take the most time)
3. Try to improve performance of bottlenecks by modifying code
4. Benchmark alternative code to identify best alternative

A few tools:

• Profiling
  • cProfile, viztracer to generate profile info
  • snakeviz to display results
  • On CARC systems, download output files to view graphics locally
• Benchmarking
  • timeit (Run a function repeatedly to get stats)
  • time (Get start and end time)

• examples/write.py
• Generate some data, write to .txt file
• Let’s measure performance for baseline
• python -m cProfile -o write.log examples/write.py -n 100
  • Write 100 files
  • Generates binary file, need utility to view
  • snakeviz write.log
• python -m cProfile -s tottime write.py -n 100
  • Generates text output, might need tools to parse

This snippet is not great

def write_data(x,y,n,t):
    filename=("output/%s%05d" %(output,i))

    i_max=len(x)
    j_max=len(y)
    data=np.zeros((i_max,j_max))

    for i in range(i_max):
        for j in range(j_max):
            data[i,j]=generate_data(x[i],y[j],nFiles,fileID)

            np.savetxt(filename,data)

• Code first, optimize later (if needed)
• Profile code to identify bottlenecks
• Simplify when possible (do less)
• Vectorize code
• Use existing solutions
• Parallelize when appropriate

• Packages like numpy often have built-in vectorized versions
• Apply same operation to each element
• Often written in C/C++/Fortran
• Use built in functions when possible

numpy.meshgrid(*xi, copy=True, sparse=False, indexing='xy')

Return coordinate matrices from coordinate vectors. Make N-D coordinate arrays for vectorized evaluations of N-D scalar/vector fields over N-D grids, given one-dimensional coordinate arrays x1, x2,…, xn.

• Usually if you’re implementing a loop there’s a faster way

def write_data(X,Y,n,t):
    filename=("output/%s%05d" %(output,i))

    Z = generate_data(X,Y,nFiles,i)
    np.savetxt(filename,Z)

Where

x = np.arange(x_origin-size/2,x_origin+size/2,1)
y = np.arange(y_origin-size/2,y_origin+size/2,1)
X,Y = np.meshgrid(x,y)

• See example under examples/write_vectorized.py

python3 -m cProfile -s tottime examples/write_vectorized.py -n 100 write_vectorized > write_vectorized

Original write.py ~4.8s

New write_vectorized.py ~ 1.4s

• Our example has the line

    np.savetxt(filename,data)

• Generate many small files in plain, text
• Sacrifice readibility for performance?

    np.save(filename,data)

• We save time by writing less data, in binary
• numpy.savetxt
• numpy.save

write_vectorized.py ~1.4s

examples/write_vectorized_binary.py ~0.46s

• Performance is pretty good BUT
• ~40% of time is WAITING on opening files
• Keep in mind for later
• Assume we have fine tuned program
• Assume it’s worth our time to make it faster
• What next?

• Key concept: speedup (decrease in runtime)
• Parallel programming is easier with one compute node
  • Using multiple nodes requires more effort
• “Parallel” can be different things
  • Data vs. task parallelism
  • “Pleasantly” parallel or “Embarassingly” parallel
    • Concurrent execution of different parts of a larger computation
    • “High throughput computing”
  • Implicit vs. explicit parallel programming

• Some computations are not worth parallelizing
• Some costs to parallelizing (overhead):
  • changing code
  • spawning child processes
  • copying data and environment
  • communications
• Speedup not proportional to number of cores (Amdahl’s law)
• Optimal number of cores
  • depends on specific computations
  • experiment to find

• Compute nodes have different configurations
  • number of cores
  • amount of memory
• On CARC systems:
  • Discovery Resource Overview
  • Enter sinfo2 in shell to see node types
  • 1 logical CPU = 1 core = 1 thread (--cpus-per-task)

#!/bin/bash

#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --mem=16GB
#SBATCH --time=1:00:00
#SBATCH --account=<account_id>

module purge
module load gcc/8.3.0
module load openblas/0.3.8
module load python/3.7.6

python3 /path/to/script.py

• Be careful mixing implicit and explicit parallelism
• Implicit parallel code may use more resources than intended
• Turn off implicit parallelism with export OMP_NUM_THREADS=1

• Multiprocessing is a built in package
• Create pool of processes
• Typical example, apply a function over multiple inputs

def f(x):
    return x*x

if __name__ == '__main__':
    with Pool(5) as p:
        print(p.map(f, [1, 2, 3]))

• Since we have multiple arguments

with mp.Pool(processes=nWorkers) as pool:
    for i in range(0,nFiles):
        pool.apply(write_data,args=(X,Y,output,nFiles,i,))

• Also add -w option for nWorkers
• See example under examples/write_multiprocessing.py

• Let’s use Viztracer to profile
• viztracer --log_multiprocess -o multiprocessing.html examples/write_multiprocessing.py -w 3 -n 100
• use ‘wasd’ to navigate

• For 1 worker, ~0.3 s

• For 3 workers, ~0.3 s

• Despite more workers, no speedup
• There is communication overhead
• This line is problematic:

pool.apply(write_data,args=(X,Y,output,nFiles,i))

• During communication, variables are “pickled” or serialized
• We pass in X,Y,output,nFiles even though they never change

• mpi4py allows you to add mpi functionality to your scripts
• Send messages across compute nodes
• Easier to scale up
  • Assuming it’s worth communication overhead
• Easier to read?

• In some cases file operations are a bottleneck
• We saw in one example ~ 40% of time was waiting to open a file
• It’s best to write one large file
• Libraries like hdf5, h5py allow multiple processes to write to same file
• Parallelization not needed for speedup

• In examples/write_vectorized_binary.py we can modifying write_data
• See examples/write_hdf5.py

def write_data(X,Y,output,nFiles,i,hf):

    filename=("output/%s%05d" %(output,i))
    Z = generate_data(X,Y,nFiles,i)

    hf.create_dataset(filename,data=Z)

• Where hf = h5py.File('output/data.h5', 'w')

examples/write_vectorized_binary.py ~5.46s (1000 writes)

examples/write_hdf5.py ~2.54s (1000 writes)

• See examples/write_final.py
• set hf=h5py.File('output/data.hdf5','w',driver='mpio',comm=comm)
• Use same write_data function as single core version
• Reading data is a bit harder
  • Use filename ‘data00123’ as ‘key’ h5py will return dataset

#!/bin/bash

#SBATCH --ntasks=16
#SBATCH --cpus-per-task=1
#SBATCH --mem-per-cpu=3GB
#SBATCH --time=1:00:00
#SBATCH --account=<account_id>

module purge
module load gcc/8.3.0
module load openblas/0.3.8
module load openmpi/4.0.2
module load pmix/3.1.3
module load python/3.7.6

srun --mpi=pmix_v2 python3 /path/to/script.py

• multiprocessig
• mpi4py
  
• h5py
• hdf5
• cProfile
• snakeviz
• viztracer

• Questions?
  • carc-support@usc.edu
• Workshop schedule